vNext: Add migration step for publication-dissertation - #920

Sorry if I have seen this a bit too late.

I think that this migration should be mandatory, transparent and done during the upgrade.
The reason is that having thesis and dissertation was an error, a duplication that no one noticed. The drawback is that the DOIs for thesis are created as custom text, instead of dissertation. This is wrong, and goes against our claim that we follow closely the DataCite schema.

The only thing we can do with the script is to check if thesis and dissertation exist in the system. If yes, we should apply it. We can provide the above explanation in the upgrade doc. If not, then we should skip it.

To be discussed.

Just to be clear, we're talking about forcing the migration from the resource type publication-thesis to publication-dissertation, right?

Frankly, in this particular instance, I disagree with having a forced migration of the values.
At least in Austria, talking about a "dissertation" typically strongly hints at PhD-level academic work, whereas "thesis" is used more generally and usually (but not exclusively) refers to a Bachelor's or Master's thesis.
For us, DataCite having a value "dissertation" but not a more general "thesis" is actually strange, because I'd argue that every dissertation is a thesis but not every thesis is a dissertation.

In our instance, we have a bunch of records that do refer to Bachelor's or Master's theses specifically [1], not a PhD dissertation.
Forcibly changing these references to "dissertation" would be semantically incorrect in our setting, or at the very least raise quite a few eyebrows and for sure earn me a few extra support requests.

In my opinion, this migration should not be mandatory, but it should be left to the instance operator to decide whether or not they want to migrate existing records to the new value.
I would suggest two questions to the operator (either interactive when running the script, or as flags):

• Do you want to create the new resource type publication-dissertation?
• If yes, do you want to migrate all existing records from publication-thesis to publication-dissertation?

I can see scenarios where instance operators might have customized the resource types beforehand already, and have explicitly decided on publication-thesis over publication-dissertation.
In this case, they might even answer the first question with a "no".
For them, InvenioRDM adhering to DataCite does not matter.

[1] e.g. https://researchdata.tuwien.ac.at/records/rjqr7-0b076

To be clear, if we were setting up a fresh instance, that'd be a very different question for us; then we might roll with the default value (and perhaps add our own custom value), but I don't intend to migrate our existing records forcibly.

DataCite metadata working group already had a considerable amount of discussions on this topic, and they came up with:
https://datacite-metadata-schema.readthedocs.io/en/4.7/appendices/appendix-1/resourceTypeGeneral/#dissertation
Description: A written essay, treatise, or thesis, especially one written by a candidate for the degree of Doctor of Philosophy.

This change changes the internal ID, but keeps the text Thesis, and removes the wrong duplication and mistake done in the past.

I see and understand that, but I still think that with all the possibilities that InvenioRDM offers and actually advertises regarding customization of controlled vocabularies, I believe that the instances' data is not ours to forcibly change anymore.
I believe that the instance operators know their own instance best, and should have the option to skip this step if desired.

Of course, the possibility of skipping should be based on informed consent and can also be on an opt-out basis.
I would propose a prominent warning either in the migration script or the upgrade docs, and information on how to opt out of this change if desired.

I would like a prominent warning in the update docs, too! Our librarians are discussing a lot, too, and not always comply with the results from Datacite discussions. I would even say, they rarely do.

I see and understand that, but I still think that with all the possibilities that InvenioRDM offers and actually advertises regarding customization of controlled vocabularies, I believe that the instances' data is not ours to forcibly change anymore. I believe that the instance operators know their own instance best, and should have the option to skip this step if desired.

Of course, the possibility of skipping should be based on informed consent and can also be on an opt-out basis. I would propose a prominent warning either in the migration script or the upgrade docs, and information on how to opt out of this change if desired.

something like that: https://github.com/inveniosoftware/invenio-app-rdm/pull/3501/changes which would be an opt in

Haha I am glad this is getting discussed! A quick summary from my end:

The situation as I understand it

Use to be

• resource type with id publication-thesis
  • with datacite_general: Text and datacite_type: Thesis
  • with eurepo: info:eu-repo/semantics/doctoralThesis (inveniosoftware/invenio-rdm-records@f7d685e#diff-b57129335e1f5af65eb0285bf5f9e4d0583597ae368fc23d1747a5bcb9b77f48L446)
• resource type with id publication-dissertation
  • with datacite_general: Dissertation and datacite_type: ""
  • eurepo: info:eu-repo/semantics/other

(eurepo just added to show surrounding semantic whether right or wrong...)

These are both "technically" datacite compliant (use available datacite_general values from the controlled list), but publication-thesis was deemed not "semantically" compliant: it was "a duplication that no one noticed", because there was already publication-dissertation. publication-thesis was seen as pretty much the same thing as publication-dissertation and Datacite says to use datacite_general Text only if no other value is applicable (and Dissertation was applicable in a sense). So publication-thesis was removed from the fixture and code was added to the migration script to align/convert records+drafts with resource type id: publication-thesis to id: publication-dissertation.

The problem with forcing that migration as I see it

I think Max put it into words better than I did. Because instances may have customized their resource types, this migration is potentially fragile and invasive:

• publication-thesis may exist, but publication-dissertation may not
• publication-thesis/publication-dissertation may have their datacite_{general,type} changed to values that don't fit (e.g., publication-thesis has the 'right' datacite values in the instance and publication-dissertation doesn't or whatever whim of instance)
• instance operators may want to keep both, because publication-dissertation is for PhD and publication-thesis is for the rest or whatever reason/desire they have. (e.g., like @wgresshoff our librarians have many thoughts on the meaning and correspondences of different types, and those change over time anyway). Looking at https://datacite-metadata-schema.readthedocs.io/_/downloads/en/4.7/pdf/ : datacite does seem to imply dissertation is a bit more for PhD for instance.

All that to say that, in general, when these highly visible and commonly customized vocabularies are changed, my thinking is that we should err on suggesting rather than imposing the changes. The flag in that PR works for me! Providing a separate script the migrator may want to run is fine too. "Whatever a framework does for you, it does to you" and all that.

Thanks!

Using a custom Text resource type should be avoided whenever possible. It makes interoperability between more difficult. This situation will happen again in the future when DataCite will include a new type in the supported resource types that we have now mapped to a custom Text. And this is happening and it will happen with other metadata fields.
If we follow the proposed logic, we will never get closer to the DataCite data model, which is one of the core part of InvenioRDM, including interoperability.

I understand the customizations of resource types, but this should be handled by the script. I don't see any technical limitation. If we simply propose or suggest it, I am pretty convinced no one will ever do it.

This will require a discussion on a call. It would be preferable to also have @tmorrell if he is available (and any other interested person).

I'm not suggesting a mere warning, I'm suggesting an informed decision point, and like I've said, I'm also fine with an opt-out.
For example, at the beginning of the script:

WARNING: We did an oopsie with the resource types! DataCite defines `publication-dissertation` as the value for *all* theses (not only, but especially dissertations) but in InvenioRDM the value is `publication-thesis`.
To align better with DataCite, this should be updated to `publication-dissertation`.

Do you want to update the resource type `publication-thesis` to `publication-dissertation` and update all affected records now? [Y/n]:

And in case the user explicitly selects "no" (the default being "yes"), print some information on how they can do this in the future.

Could also be prominently documented in the upgrade guide, but I think the script would be a better place.
Because the script is probably the closest thing to a "must" for instance operators to go through, as opposed to reading the docs carefully.
So basically in full agreement here:

If we simply propose or suggest it, I am pretty convinced no one will ever do it.

InvenioRDM is interoperable via API, but instances are generally not interoperable on a per-entry basis.
For instance, if one instance does and another doesn't import the ROR funders, it's quite easy to arrive at a point where records cannot be transferred between the instances without some data massaging anyway.

Regarding metadata schema, from my understanding we're only "heavily inspired" by DataCite but don't follow it 100% either.
Which is fine, as long as we have a good enough mapping to/from DataCite, which we do.

I'd also vote for making the migration optional. Resource types are really specific to the institution, and I think there are valid use cases to having both resource types. I like @max-moser 's suggestion of having an opt-out as part of the upgrade script.

@fenekku @max-moser I have added a new commit to explain the why.
The user needs to understand the reasons, which are mainly related to our choice to follow closely the DataCite metadata schema and enhance interoperability.

v14 cleanup
Re-assigned to @slint (can delegate to Saksham too)

Per maintainer meeting, a section dedicated to the optional publication-dissertation migration will be added at the end of the upgrade-v14.0.md (and can be mentioned in version-v14.0.md). There, we will explain the change in the original resource type listing and provide a script to run for upgraders that kept the same resource types and would also like the same change applied to their instance. The script will also serve as a starting point for upgraders who don't have the exact same resource types but want to carry out a similar update (e.g. change the datacite_general value to "Dissertation" for one of their resource type. It will highlight how InvenioRDM's respect for instance data customizations and choices makes this an optional step, doable at a later time when upgrader's instance's stakeholders have been consulted.

Ping @fenekku, I've pushed the changes we've discussed and you've summarized in #920 (comment). If you can have a quick look and make sure the placement and wording are fine, feel free also to apply any small changes directly and merge.

I know you're making a full pass as well, so I'm leaving it up to you to make your merge/rebase life easier 😅

@fenekku thanks for all the suggestions, I've applied and pushed. Good to merge now :)